Skip to content

Conversation

@shefeek-jinnah
Copy link
Collaborator

@shefeek-jinnah shefeek-jinnah commented Dec 8, 2025

Summary

Adds configurable time-to-live (TTL) based caching for snapshot resolution in DuckLakeCatalog, allowing users to balance data freshness against query performance.

Changes

  • Added SnapshotConfig struct with ttl_seconds: Option<u64> field
  • Added DuckLakeCatalog::new_with_config() for custom snapshot configuration
  • Implemented three caching modes:
    • TTL = 0 (default): Always query latest snapshot
    • TTL > 0: Cache snapshot for specified duration
    • TTL = None: Cache snapshot forever
  • Thread-safe implementation using RwLock with double-check locking

API

// Default: always fresh (TTL=0)
let catalog = DuckLakeCatalog::new(provider)?;

// Custom TTL: cache for 5 seconds
let config = SnapshotConfig { ttl_seconds: Some(5) };
let catalog = DuckLakeCatalog::new_with_config(provider, config)?;

// Cache forever
let config = SnapshotConfig { ttl_seconds: None };
let catalog = DuckLakeCatalog::new_with_config(provider, config)?;

Testing

- 5 new unit tests covering all TTL modes and concurrent access
- All existing tests pass (82 total)

@shefeek-jinnah shefeek-jinnah marked this pull request as draft December 8, 2025 06:46
@zfarrell
Copy link
Contributor

zfarrell commented Dec 8, 2025

Noting the logic bug we discussed on the call: it's currently possible in all caching configurations other than none for a snapshot to change during a user's query and to have the user query be served from multiple snapshots. We should address this so that only one snapshot id is used for the user's query.

@shefeek-jinnah shefeek-jinnah marked this pull request as ready for review December 9, 2025 11:37
@zfarrell
Copy link
Contributor

@shefeek-crypto - should we go ahead and close this (in favor of #6), or is there anything here that you're hoping to keep?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants